Controlling Robots in Web Search Engines

Robots are deployed by a Web search engine for collecting information from diierent Web servers in order to maintain the currency of its data base of Web pages. In this paper, we investigate the number of robots to be used by a search engine so as to maximize the currency of the data base without putting an unnecessary load on the network. We use a queueing model to represent the system. The arrivals to the queueing system are Web pages brought by the robots; service corresponds to the indexing of these pages. The objective is to nd the number of robots, and thus the arrival rate of the queueing system, such that the indexing queue is neither starved nor saturated. For this, we consider a nite-buuer queueing system and deene the cost function to be minimized as a weighted sum of the loss rate and the fraction of time that the system is empty. Both static and dynamic policies are considered. In the static setting the number of robots is held xed; in the dynamic setting robots may be reactivated/desactivated at some particular points in time. Under the assumption that arrivals form a Poisson process, and that service times are independent and identically distributed random variables with an exponential distribution, we determine the optimal number of robots to deploy both in the static and in the dynamic setting. Numerical results indicate that substential gains can be achieved by dynamically controlling the activity of the robots.